Method and apparatus for intra-prediction coding of video data

ABSTRACT

An apparatus for decoding video data includes a decoder configured to decode, from a bitstream, a syntax element indicating an intra-prediction type of a current block of the video data, and an intra-predictor configured to generate a prediction block for the current block by selectively performing matrix based intra-prediction (MIP) or regular intra-prediction based on the intra-prediction type of the current block indicated by the syntax element.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims under 35 U.S.C. § 119(a) the benefit of KoreanPatent Application No. 10-2019-0082130 filed on Jul. 8, 2019, KoreanPatent Application No. 10-2019-0102494 filed on Aug. 21, 2019, KoreanPatent Application No. 10-2019-0102495 filed on Aug. 21, 2019, KoreanPatent Application No. 10-2019-0123492 filed on Oct. 6, 2019, and KoreanPatent Application No. 10-2020-0083979 filed on Jul. 8, 2020, the entirecontents of which are incorporated herein by reference.

BACKGROUND (a) Technical Field

The present disclosure relates to encoding and decoding of video data.

(b) Description of the Related Art

Since the volume of video data typically is larger than that of voicedata or still image data, storing or transmitting video data withoutprocessing for compression requires a significant amount of hardwareresources including memory.

Accordingly, in storing or transmitting video data, the video data isgenerally compressed using an encoder so as to be stored or transmitted.Then, a decoder receives the compressed video data, and decompresses andreproduces the video data. Compression techniques for video includeH.264/AVC and High Efficiency Video Coding (HEVC), which improves codingefficiency over H.264/AVC by about 40%.

However, for video data, picture size, resolution, and frame rate aregradually increasing, and accordingly the amount of data to be encodedis also increasing. Accordingly, a new compression technique havingbetter encoding efficiency and higher image quality than the existingcompression technique is required.

SUMMARY

The present disclosure discloses improved techniques forintra-prediction coding of a block of video data.

In accordance with one aspect of the present disclosure, a method ofdecoding video data includes decoding, from a bitstream, a syntaxelement indicating an intra-prediction type of a current block of thevideo data, the intra-prediction type including matrix basedintra-prediction (MIP) and regular intra-prediction; and generating aprediction block for the current block by selectively performing the MIPor the regular intra-prediction based on the intra-prediction type ofthe current block indicated by the syntax element.

In generating the prediction block for the current block by performingthe MIP, the method further includes decoding, from the bitstream, asyntax element indicating an MIP mode for the current block, the syntaxelement being represented as a truncated binary code specifying one of aplurality of MIP prediction modes allowed for a width and a height ofthe current block; deriving an input boundary vector using neighboringsamples adjacent to the current block based on the width and the heightof the current block; generating predicted samples for the current blockbased on matrix-vector multiplication between the input boundary vectorand a matrix predefined for the MIP mode; and deriving the predictionblock for the current block based on the predicted samples.

In generating the prediction block for the current block by performingthe regular intra-prediction, the method further includes deriving MostProbable Mode (MPM) candidates based on a regular intra-prediction modeof neighboring blocks adjacent to the current block and configuring anMPM list for the current block; and deriving a regular intra-predictionmode for the current block based on the MPM list. When anintra-prediction type of the neighboring blocks is the MIP, the regularintra-prediction mode of the neighboring block is set (regarded) as aPLANAR mode.

In accordance with another aspect of the present disclosure, anapparatus for decoding video data includes a decoder configured todecode, from a bitstream, a syntax element indicating anintra-prediction type of a current block of the video data, theintra-prediction type including matrix based intra-prediction (MIP) andregular intra-prediction; and an intra-predictor configured to generatea prediction block for the current block by selectively performing theMIP or the regular intra-prediction based on the intra-prediction typeof the current block indicated by the syntax element.

In generating the prediction block for the current block by performingthe MIP, the intra-predictor is configured to decode, from thebitstream, a syntax element indicating an MIP mode for the currentblock, the syntax element being represented as a truncated binary codespecifying one of a plurality of MIP prediction modes allowed for awidth and a height of the current block; derive an input boundary vectorusing neighboring samples adjacent to the current block based on thewidth and the height of the current block; generate predicted samplesfor the current block based on matrix-vector multiplication between theinput boundary vector and a matrix predefined for the MIP mode; andderive the prediction block for the current block based on the predictedsamples.

In generating the prediction block for the current block by performingthe regular intra-prediction, the intra-predictor is configured toderive Most Probable Mode (MPM) candidates based on a regularintra-prediction mode of neighboring blocks adjacent to the currentblock and configure an MPM list for the current block; and derive aregular intra-prediction mode for the current block based on the MPMlist. In deriving the MPM candidates, when an intra-prediction type ofthe neighboring blocks is the MIP, the intra-predictor sets (regards)the regular intra-prediction mode of the neighboring block as a PLANARmode.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an exemplary block diagram of a video encoding apparatuscapable of implementing the techniques of the present disclosure.

FIG. 2 exemplarily shows a block partitioning structure using a QTBTTTstructure.

FIG. 3A exemplarily shows a plurality of intra-prediction modes.

FIG. 3B exemplarily shows a plurality of intra prediction modesincluding wide-angle intra prediction modes.

FIG. 4 is an exemplary block diagram of a video decoding apparatuscapable of implementing the techniques of the present disclosure.

FIG. 5 is a conceptual diagram illustrating the main process of MIPtechnology that may be used in the techniques of the present disclosure.

FIG. 6 illustrates smoothing filtering and interpolation filtering forconstructing reference samples.

FIGS. 7A to 7C are conceptual diagrams illustrating an exemplary methodof constructing a boundary vector that is input to matrix-vectormultiplication using left neighboring samples.

FIGS. 8A to 8C are conceptual diagrams illustrating an exemplary methodof constructing a boundary vector that is input to a matrix-vectormultiplication operation using above neighboring samples.

FIG. 9 is a flowchart illustrating a method of decoding video dataaccording to an embodiment of the present disclosure.

DETAILED DESCRIPTION

Hereinafter, some embodiments of the present disclosure will bedescribed in detail with reference to the accompanying drawings. Itshould be noted that, in adding reference numerals to the constituentelements in the respective drawings, like reference numerals designatelike elements, although the elements are shown in different drawings.Further, in the following description of the present disclosure, adetailed description of known functions and configurations incorporatedherein will be omitted to avoid obscuring the subject matter of thepresent disclosure.

FIG. 1 is an exemplary block diagram of a video encoding apparatuscapable of implementing the techniques of the present disclosure.Hereinafter, a video encoding apparatus and elements of the apparatuswill be described with reference to FIG. 1 .

The video encoding apparatus includes a picture splitter 110, apredictor 120, a subtractor 130, a transformer 140, a quantizer 145, arearrangement unit 150, an entropy encoder 155, an inverse quantizer160, an inverse transformer 165, an adder 170, a filter unit 180, and amemory 190.

Each element of the video encoding apparatus may be implemented inhardware or software, or a combination of hardware and software. Thefunctions of the respective elements may be implemented as software, anda microprocessor may be implemented to execute the software functionscorresponding to the respective elements.

One video includes a plurality of pictures. Each picture is split into aplurality of regions, and encoding is performed on each region. Forexample, one picture is split into one or more tiles or/and slices.Here, the one or more tiles may be defined as a tile group. Each tile orslice is split into one or more coding tree units (CTUs). Each CTU issplit into one or more coding units (CUs) by a tree structure.Information applied to each CU is encoded as a syntax of the CU, andinformation applied to CUs included in one CTU in common is encoded as asyntax of the CTU. In addition, information applied to all blocks in oneslice in common is encoded as a syntax of a slice header, andinformation applied to all blocks constituting a picture is encoded in apicture parameter set (PPS) or a picture header. Furthermore,information which a plurality of pictures refers to in common is encodedin a sequence parameter set (SPS). In addition, information referred toby one or more SPSs in common is encoded in a video parameter set (VPS).Information applied to one tile or tile group in common may be encodedas a syntax of a tile or tile group header.

The picture splitter 110 is configured to determine the size of a codingtree unit (CTU). Information about the size of the CTU (CTU size) isencoded as a syntax of the SPS or PPS and is transmitted to the videodecoding apparatus.

The picture splitter 110 is configured to split each pictureconstituting the video into a plurality of CTUs having a predeterminedsize, and then recursively split the CTUs using a tree structure. In thetree structure, a leaf node serves as a coding unit (CU), which is abasic unit of coding.

The tree structure may be a QuadTree (QT), in which a node (or parentnode) is split into four sub-nodes (or child nodes) of the same size, aBinaryTree (BT), in which a node is split into two sub-nodes, aTernaryTree (TT), in which a node is split into three sub-nodes at aratio of 1:2:1, or a structure formed by a combination of two or more ofthe QT structure, the BT structure, and the TT structure. For example, aQuadTree plus BinaryTree (QTBT) structure may be used, or a QuadTreeplus BinaryTree TernaryTree (QTBTTT) structure may be used. Here, BTTTmay be collectively referred to as a multiple-type tree (MTT).

FIG. 2 exemplarily shows a QTBTTT splitting tree structure. As shown inFIG. 2 , a CTU may be initially split in the QT structure. The QTsplitting may be repeated until the size of the splitting block reachesthe minimum block size MinQTSize of a leaf node allowed in the QT. Afirst flag (QT_split_flag) indicating whether each node of the QTstructure is split into four nodes of a lower layer is encoded by theentropy encoder 155 and signaled to the video decoding apparatus. Whenthe leaf node of the QT is not larger than the maximum block size(MaxBTSize) of the root node allowed in the BT, it may be further splitinto one or more of the BT structure or the TT structure. The BTstructure and/or the TT structure may have a plurality of splittingdirections. For example, there may be two directions, namely, adirection in which a block of a node is horizontally split and adirection in which the block is vertically split. As shown in FIG. 2 ,when MTT splitting is started, a second flag (mtt_split_flag) indicatingwhether nodes are split, a flag indicating a splitting direction(vertical or horizontal) in the case of splitting, and/or a flagindicating a splitting type (Binary or Ternary) are encoded by theentropy encoder 155 and signaled to the video decoding apparatus.Alternatively, prior to encoding the first flag (QT_split_flag)indicating whether each node is split into 4 nodes of a lower layer, aCU splitting flag (split_cu_flag) indicating whether the node is splitmay be encoded. When the value of the CU split flag (split_cu_flag)indicates that splitting is not performed, the block of the node becomesa leaf node in the splitting tree structure and serves a coding unit(CU), which is a basic unit of encoding. When the value of the CU splitflag (split_cu_flag) indicates that splitting is performed, the videoencoding apparatus starts encoding the flags in the manner describedabove, starting with the first flag.

When QTBT is used as another example of a tree structure, there may betwo splitting types, which are a type of horizontally splitting a blockinto two blocks of the same size (i.e., symmetric horizontal splitting)and a type of vertically splitting a block into two blocks of the samesize (i.e., symmetric vertical splitting). A split flag (split_flag)indicating whether each node of the BT structure is split into block ofa lower layer and splitting type information indicating the splittingtype are encoded by the entropy encoder 155 and transmitted to the videodecoding apparatus. There may be an additional type of splitting a blockof a node into two asymmetric blocks. The asymmetric splitting type mayinclude a type of splitting a block into two rectangular blocks at asize ratio of 1:3, or a type of diagonally splitting a block of a node.

CUs may have various sizes according to QTBT or QTBTTT splitting of aCTU. Hereinafter, a block corresponding to a CU (i.e., a leaf node ofQTBTTT) to be encoded or decoded is referred to as a “current block.” AsQTBTTT splitting is employed, the shape of the current block may besquare or rectangular.

The predictor 120 is configured to predict the current block to generatea prediction block. The predictor 120 includes an intra-predictor 122and an inter-predictor 124.

In general, each of the current blocks in a picture may be predictivelycoded. In general, prediction of a current block is performed using anintra-prediction technique (using data from a picture containing thecurrent block) or an inter-prediction technique (using data from apicture coded before a picture containing the current block). Theinter-prediction includes both unidirectional prediction andbi-directional prediction.

The intra-prediction unit 122 is configured to predict pixels in thecurrent block using pixels (reference pixels) positioned around thecurrent block in the current picture including the current block. Thereis a plurality of intra-prediction modes according to the predictiondirections. For example, as shown in FIG. 3 , the plurality ofintra-prediction modes may include two non-directional modes, whichinclude a PLANAR mode and a DC mode, and 65 directional modes.Neighboring pixels and an equation to be used are defined differentlyfor each prediction mode. The table below lists intra-prediction modenumbers and names thereof.

TABLE 1 Intra prediction mode Associated name 0 INTRA_PLANAR 1 INTRA_DC2 . . . 66 INTRA_ANGULAR2 . . . INTRA_ANGULAR66

For efficient directional prediction for a rectangular-shaped currentblock, directional modes (intra-prediction modes 67 to 80 and −1 to −14)indicated by dotted arrows in FIG. 3B may be additionally used. Thesemodes may be referred to as “wide angle intra-prediction modes.” In FIG.3B, arrows indicate corresponding reference samples used for prediction,not indicating prediction directions. The prediction direction isopposite to the direction indicated by an arrow. A wide-angle intraprediction mode is a mode in which prediction is performed in adirection opposite to a specific directional mode without additional bittransmission when the current block has a rectangular shape. In thiscase, among the wide angle intra-prediction modes, some wide angleintra-prediction modes available for the current block may be determinedbased on a ratio of a width and a height of the rectangular currentblock. For example, wide angle intra-prediction modes with an angle lessthan 45 degrees (intra prediction modes 67 to 80) may be used when thecurrent block has a rectangular shape with a height less than the widththereof. Wide angle intra-prediction modes with an angle greater than−135 degrees (intra-prediction modes −1 to −14) may be used when thecurrent block has a rectangular shape with the height greater than thewidth thereof.

The intra-predictor 122 may determine an intra-prediction mode to beused in encoding the current block. In some examples, theintra-predictor 122 may encode the current block using severalintra-prediction modes and select an appropriate intra-prediction modeto use from the tested modes. For example, the intra-predictor 122 maycalculate rate distortion values using rate-distortion analysis ofseveral tested intra-prediction modes, and may select anintra-prediction mode that has the best rate distortion characteristicsamong the tested modes.

The intra-predictor 122 is configured to select one intra-predictionmode from among the plurality of intra-prediction modes, and predictsthe current block using neighboring pixels (reference pixels) and anequation determined according to the selected intra-prediction mode.Information about the selected intra-prediction mode is encoded by theentropy encoder 155 and transmitted to the video decoding apparatus.

In addition, the intra-predictor 122 may generate a prediction block forthe current block, using matrix-based intra-prediction (MIP), which willbe described later. The intra-predictor 122 generates a prediction blockfor the current block using a boundary vector derived from samplesreconstructed on the left side of the current block and samplesreconstructed above the current block, a predefined matrix, and anoffset vector.

The inter-predictor 124 is configured to generate a prediction block forthe current block through motion compensation. The inter-predictor 124may search for a block most similar to the current block in a referencepicture which has been encoded and decoded earlier than the currentpicture, and generate a prediction block for the current block using thesearched block. Then, the inter-predictor is configured to generate amotion vector corresponding to a displacement between the current blockin the current picture and the prediction block in the referencepicture. In general, motion estimation is performed on a luma component,and a motion vector calculated based on the luma component is used forboth the luma component and the chroma component. The motion informationincluding information about the reference picture and information aboutthe motion vector used to predict the current block is encoded by theentropy encoder 155 and transmitted to the video decoding apparatus.

The subtractor 130 is configured to subtract the prediction blockgenerated by the intra-predictor 122 or the inter-predictor 124 from thecurrent block to generate a residual block.

The transformer 140 may split the residual block into one or moretransform blocks, and applies the transformation to the one or moretransform blocks, thereby transforming the residual values of thetransform blocks from the pixel domain to the frequency domain. In thefrequency domain, the transformed blocks are referred to as coefficientblocks containing one or more transform coefficient values. Atwo-dimensional transform kernel may be used for transformation, andone-dimensional transform kernels may be used for horizontaltransformation and vertical transformation, respectively. The transformkernels may be based on a discrete cosine transform (DCT), a discretesine transform (DST), or the like.

The transformer 140 may transform residual signals in the residual blockusing the entire size of the residual block as a transformation unit. Inaddition, the transformer 140 may partition the residual block into twosub-blocks in a horizontal or vertical direction, and may transform onlyone of the two sub-blocks. Accordingly, the size of the transform blockmay be different from the size of the residual block (and thus the sizeof the prediction block). Non-zero residual sample values may not bepresent or may be very rare in the untransformed subblock. The residualsamples of the untransformed subblock are not signaled, and may beregarded as “0” by the video decoding apparatus. There may be multiplepartition types according to the partitioning direction and partitioningratio. The transformer 140 may provide information about the coding mode(or transform mode) of the residual block (e.g., information indicatingwhether the residual block is transformed or the residual subblock istransformed, and information indicating the partition type selected topartition the residual block into subblocks, and information identifyinga subblock that is transformed is performed) to the entropy encoder 155.The entropy encoder 155 may encode the information about the coding mode(or transform mode) of the residual block.

The quantizer 145 is configured to quantize transform coefficientsoutput from the transformer 140, and output the quantized transformcoefficients to the entropy encoder 155. For some blocks or frames, thequantizer 145 may directly quantize a related residual block withouttransformation.

The rearrangement unit 150 may reorganize the coefficient values for thequantized residual value. The rearrangement unit 150 may change the2-dimensional array of coefficients into a 1-dimensional coefficientsequence through coefficient scanning. For example, the rearrangementunit 150 may scan coefficients from a DC coefficient to a coefficient ina high frequency region using a zig-zag scan or a diagonal scan tooutput a 1-dimensional coefficient sequence. Depending on the size ofthe transformation unit and the intra-prediction mode, a vertical scan,in which a two-dimensional array of coefficients is scanned in a columndirection, or a horizontal scan, in which two-dimensional block-shapedcoefficients are scanned in a row direction, may be used instead of thezig-zag scan. That is, a scan mode to be used may be determined amongthe zig-zag scan, the diagonal scan, the vertical scan and thehorizontal scan according to the size of the transformation unit and theintra-prediction mode.

The entropy encoder 155 is configured to encode the one-dimensionalquantized transform coefficients output from the rearrangement unit 150using various encoding techniques such as Context-based Adaptive BinaryArithmetic Code (CABAC) and exponential Golomb, to generate a bitstream.

The entropy encoder 155 may encode information such as a CTU size, a CUsplit flag, a QT split flag, an MTT splitting type, and an MTT splittingdirection, which are associated with block splitting, such that thevideo decoding apparatus may split the block in the same manner as inthe video encoding apparatus. In addition, the entropy encoder 155 mayencode information about a prediction type indicating whether thecurrent block is encoded by intra-prediction or inter-prediction, andencode intra-prediction information (i.e., information about anintra-prediction mode) or inter-prediction information (informationabout a reference picture index and a motion vector) according to theprediction type.

The inverse quantizer 160 is configured to inversely quantize thequantized transform coefficients output from the quantizer 145 togenerate transform coefficients. The inverse transformer 165 isconfigured to transform the transform coefficients output from theinverse quantizer 160 from the frequency domain to the spatial domainand reconstructs the residual block.

The adder 170 is configured to add the reconstructed residual block tothe prediction block generated by the predictor 120 to reconstruct thecurrent block. The pixels in the reconstructed current block are used asreference pixels in performing intra-prediction of a next block.

The filter unit 180 is configured to filter the reconstructed pixels toreduce blocking artifacts, ringing artifacts, and blurring artifactsgenerated due to block-based prediction and transformation/quantization.The filter unit 180 may include a deblocking filter 182 and a pixeladaptive offset (SAO) filter 184.

The deblocking filter 182 is configured to filter the boundary betweenthe reconstructed blocks to remove blocking artifacts caused byblock-by-block coding/decoding, and the SAO filter 184 is configured toperform additional filtering on the deblocking-filtered video. The SAOfilter 184 is a filter used to compensate for a difference between areconstructed pixel and an original pixel caused by lossy coding.

The reconstructed blocks filtered through the deblocking filter 182 andthe SAO filter 184 are stored in the memory 190. Once all blocks in onepicture are reconstructed, the reconstructed picture may be used as areference picture for inter-prediction of blocks in a picture to beencoded next.

FIG. 4 is an exemplary functional block diagram of a video decodingapparatus capable of implementing the techniques of the presentdisclosure. Hereinafter, the video decoding apparatus and elements ofthe apparatus will be described with reference to FIG. 4 .

The video decoding apparatus may include an entropy decoder 410, arearrangement unit 415, an inverse quantizer 420, an inverse transformer430, a predictor 440, an adder 450, a filter unit 460, and a memory 470.

Similar to the video encoding apparatus of FIG. 1 , each element of thevideo decoding apparatus may be implemented in hardware, software, or acombination of hardware and software. Further, the function of eachelement may be implemented in software, and the microprocessor may beimplemented to execute the function of software corresponding to eachelement.

The entropy decoder 410 is configured to determine a current block to bedecoded by decoding a bitstream generated by the video encodingapparatus and extracting information related to block splitting, andextract prediction information and information about a residual signal,and the like required to reconstruct the current block.

The entropy decoder 410 is configured to extract information about theCTU size from the sequence parameter set (SPS) or the picture parameterset (PPS), determine the size of the CTU, and split a picture into CTUsof the determined size. Then, the decoder is configured to determine theCTU as the uppermost layer, that is, the root node of a tree structure,and extract splitting information about the CTU to split the CTU usingthe tree structure.

For example, when the CTU is split using a QTBTTT structure, a firstflag (QT_split_flag) related to splitting of the QT is extracted tosplit each node into four nodes of a sub-layer. For a node correspondingto the leaf node of the QT, the second flag (MTT_split_flag) andinformation about a splitting direction (vertical/horizontal) and/or asplitting type (binary/ternary) related to the splitting of the MTT areextracted to split the corresponding leaf node in the MTT structure.Thereby, each node below the leaf node of QT is recursively split in aBT or TT structure.

As another example, when a CTU is split using the QTBTTT structure, a CUsplit flag (split_cu_flag) indicating whether to split a CU may beextracted. When the corresponding block is split, the first flag(QT_split_flag) may be extracted. In the splitting operation, zero ormore recursive MTT splitting may occur for each node after zero or morerecursive QT splitting. For example, the CTU may directly undergo MTTsplitting without the QT splitting, or undergo only QT splittingmultiple times.

As another example, when the CTU is split using the QTBT structure, thefirst flag (QT_split_flag) related to QT splitting is extracted, andeach node is split into four nodes of a lower layer. Then, a split flag(split_flag) indicating whether a node corresponding to a leaf node ofQT is further split in the BT and the splitting direction informationare extracted.

Once the current block to be decoded is determined through splitting inthe tree structure, the entropy decoder 410 is configured to extractinformation about a prediction type indicating whether the current blockis intra-predicted or inter-predicted. When the prediction typeinformation indicates intra-prediction, the entropy decoder 410 isconfigured to extract a syntax element for the intra-predictioninformation (intra-prediction mode) for the current block. When theprediction type information indicates inter-prediction, the entropydecoder 410 is configured to extract a syntax element for theinter-prediction information, that is, information indicating a motionvector and a reference picture referred to by the motion vector.

The entropy decoder 410 is configured to extract information about thecoding mode of the residual block (e.g., information about whether theresidual block is encoded only a subblock of the residual block isencoded, information indicating the partition type selected to partitionthe residual block into subblocks, information identifying the encodedresidual subblock, quantization parameters, etc.) from the bitstream.The entropy decoder 410 also is configured to extract information aboutquantized transform coefficients of the current block as informationabout the residual signal.

The rearrangement unit 415 may change the sequence of theone-dimensional quantized transform coefficients entropy-decoded by theentropy decoder 410 to a 2-dimensional coefficient array (i.e., block)in a reverse order of the coefficient scanning performed by the videoencoding apparatus.

The inverse quantizer 420 is configured to inversely quantize thequantized transform coefficients. The inverse transformer 430 isconfigured to inversely transform the inversely quantized transformcoefficients from the frequency domain to the spatial domain based oninformation about the coding mode of the residual block to reconstructresidual signals, thereby generating a reconstructed residual block forthe current block

When the information about the coding mode of the residual blockindicates that the residual block of the current block has been coded bythe video encoding apparatus, the inverse transformer 430 uses the sizeof the current block (and thus the size of the residual block to bereconstructed) as a transform unit for the inverse quantized transformcoefficients to perform inverse transform to generate a reconstructedresidual block for the current block.

When the information about the coding mode of the residual blockindicates that only one subblock of the residual block has been coded bythe video encoding apparatus, the inverse transformer 430 uses the sizeof the transformed subblock as a transform unit for the inversequantized transform coefficients to perform inverse transform toreconstruct the residual signals for the transformed subblock, and fillsthe residual signals for the untransformed subblock with a value of “0”to generate a reconstructed residual block for the current block.

The predictor 440 may include an intra-predictor 442 and aninter-predictor 444. The intra-predictor 442 is activated when theprediction type of the current block is intra-prediction, and theinter-predictor 444 is activated when the prediction type of the currentblock is inter-prediction.

The intra-predictor 442 is configured to determine an intra-predictionmode of the current block among a plurality of intra-prediction modesbased on the syntax element for the intra-prediction mode extracted fromthe entropy decoder 410, and predict the current block using thereference pixels around the current block according to theintra-prediction mode. In addition, the intra-predictor 442 may generatea prediction block for the current block, using matrix-basedintra-prediction (MIP), which will be described later. Theintra-predictor 422 may generate a prediction block for the currentblock using a boundary vector derived from samples reconstructed on theleft side of the current block and samples reconstructed at the above ofthe current block, and a predefined matrix and offset vector.

The inter-predictor 444 is configured to determine a motion vector ofthe current block and a reference picture referred to by the motionvector using the syntax element for the intra-prediction mode extractedfrom the entropy decoder 410, and predict the current block based on themotion vector and the reference picture.

The adder 450 is configured to reconstruct the current block by addingthe residual block output from the inverse transformer and theprediction block output from the inter-predictor or the intra-predictor.The pixels in the reconstructed current block are used as referencepixels in intra-predicting a block to be decoded next.

The filter unit 460 may include a deblocking filter 462 and an SAOfilter 464. The deblocking filter 462 is configured to deblock-filterthe boundary between the reconstructed blocks to remove blockingartifacts caused by block-by-block decoding. The SAO filter 464 canperform additional filtering on the reconstructed block after deblockingfiltering to corresponding offsets so as to compensate for a differencebetween the reconstructed pixel and the original pixel caused by lossycoding. The reconstructed block filtered through the deblocking filter462 and the SAO filter 464 is stored in the memory 470. When all blocksin one picture are reconstructed, the reconstructed picture is used as areference picture for inter-prediction of blocks in a picture to beencoded next.

The techniques of the present disclosure generally are related tointra-prediction coding. The following description is mainly focused ondecoding techniques, that is, the operations of the video decoder. Theencoding techniques are briefly described because they are opposite tothe decoding techniques that are comprehensively described.

In the discussion of the Next-generation Video Coding standard (i.e.,Versatile Video Coding (VVC)), several new coding tools enabling bettercoding performance than the High Efficiency Video Coding (HEVC) havebeen introduced.

Unlike HEVC, which can use only a closest reference sample line in intraprediction, VVC can use two additional reference lines, which is knownas multiple reference line (MRL) intra prediction. The additionalreference lines can be used only for MPM modes, and cannot use non-MPMmodes. When the video encoder performs prediction for each ofdirectional modes, the encoder may select one reference line from amongthe three reference lines based on the RD COST. The index (mrl_idx) ofthe selected reference line is signaled separately and transmitted tothe video decoder.

ISP (Intra Sub-Partitions) is a coding tool that divides a CU into aplurality of subblocks of the same size in a vertical or horizontaldirection according to the size thereof, and performs prediction foreach subblock in the same intra-prediction mode. The reconstructedsample values of each subblock are available for prediction of the nextsubblock, which is processed iteratively for each subblock. The minimumblock size applicable to the ISP is 4×8 or 8×4. When the size of a blockis 4×8 or 8×4, it is partitioned into two subblocks. When the size of ablock is larger than this size, it may be partitioned into foursubblocks. When the MRL index of the block is not 0, the ISP is notused.

Unlike general in-loop filtering applied directly to reconstructed videoimages, LMCS (Luma Mapping with Chroma Scaling) adjusts the distributionof codewords for a dynamic range of a video signal to enable efficientprediction and quantization, thereby improving the coding performanceand the image quality. LMCS includes luma component mapping and chromacomponent scaling.

Luma mapping refers to in-loop mapping in which codewords for a dynamicrange of an input luma signal is redistributed into codewords capable ofimproving coding performance. Luma signal mapping may be performedthrough forward mapping and backward mapping corresponding thereto.Forward mapping divides the existing dynamic range into 16 equalsections, and then redistributes the codeword of the input video througha linear model for each section. Chroma scaling modifies a chroma signalaccording to a correlation between a luma signal and a correspondingchroma signal.

Matrix-based Intra-prediction (MIP) is a new intra-prediction techniqueintroduced in VTM 5.0. The original idea is to use a neuralnetwork-based intra-prediction technique, that is, to use a multilayerneural network to predict current PU pixel values based on adjacentreconstructed pixels. However, due to the high complexity of theprediction method using the neural network, an intra-predictiontechnique based on affine linear transform using pre-trained matriceshas been introduced.

To predict a rectangular block PU with a width W and a height H, the MIPtakes as inputs H reconstructed samples on the left of the block and Wreconstructed samples on the above of the block. The final predictedpixels are obtained by averaging, matrix-vector multiplication, linearinterpolation, and the like.

The sizes of blocks to which MIP is applied are classified into threecategories as follows.

${id{x\left( {W,H} \right)}} = \left\{ \begin{matrix}0 & {{{for}\mspace{14mu} W} = {H = 4}} \\{1\ } & {{{for}\mspace{14mu}{\max\left( {W,H} \right)}} = 8} \\2 & {{{for}\mspace{14mu}\max\left( {W,H} \right)} > 8}\end{matrix} \right.$

According to idx(W,H), the number of MIP modes (numModes), boundary size(boundarySize), and prediction block size (predW, predH, predC) aredefined as follows. In the table below, MipSizeId=idx(W,H).

TABLE 2 MipSizeId numModes boundarySize predW predH predC 0 35 2 4 4 4 119 4 4 4 4 2 11 4 Min(nTbW, 8) Min(nTbH, 8) 8

FIG. 5 is a conceptual diagram illustrating the main processes of MIPtechnology that may be used in the techniques of the present disclosure.

(1) Averaging

The main purpose of this process is to normalize the reference samples.Depending on the block size and shape (width and height) (i.e.,MipSizeId), 4 or 8 samples are obtained. When both the width and heightof the current block are 4 (i.e., W=H=4), 4 samples in total, including2 from the left and 2 from the above, are obtained (boundarySize=2). Inthe other case, 8 samples in total, including 4 from the left and 4 fromthe above, are obtained (boundarySize=4).

As shown in FIG. 5 , the above neighboring samples are denoted bybdry^(top) and the left neighboring samples are denoted by bdry^(left).By performing the averaging on bdry^(top) and bdry^(left), respectively,down-sampled sample sets bdry_(red) ^(top) and bdry_(red) ^(left) areobtained. The averaging is a downsampling process as follows.redS[x]=(Σ_(i=0) ^(bDwn-1)refS[x*bDwn+i]+(1<<(Log 2(bDwn)−1)))>>Log2(bDwn)

In the equation above, bDwn denotes a downsampling scale value(nTbs/boundarySize), and refS denotes an original reference sample. Thecalculated redS is stored as bdry_(red) ^(left) for the left neighborsand as bdry_(red) ^(top) for the above neighbors.

The down-sampled reference samples are stitched into a vector of length4 or 8. The reduced boundary vector bdry_(red) which is input to thevector-matrix multiplication is defined as the equation below. Forexample, when W=H=4 and the MIP mode is less than 18, the boundaryvector is constructed by stitching in order of bdry_(red) ^(left) andbdry_(red) ^(top). When W=H=4 and the MIP mode is greater than or equalto 18, they are stitched in order of bdry_(red) ^(left) and bdry_(red)^(top). In the following equation, “mode” denotes the MIP mode.

${bdry}_{red} = \left\{ \begin{matrix}\left\lbrack {{bdry_{red}^{top}}\ ,{{bdr}y_{red}^{left}}} \right\rbrack & {{{for}\mspace{14mu} W} = {H = {{4\mspace{14mu}{and}\mspace{14mu}{mode}} < 18}}} \\\left\lbrack {{bdry_{red}^{left}},{bdry}_{red}^{top}} \right\rbrack & {{{for}\mspace{14mu} W} = {H = {{4\mspace{14mu}{and}\mspace{14mu}{mode}} \geq 18}}} \\\left\lbrack {{bdry}_{red}^{top}\ ,{{bdr}y_{red}^{left}}} \right\rbrack & {{{for}\mspace{14mu}{\max\left( {W,H} \right)}} = {{8\mspace{14mu}{and}\mspace{14mu}{mode}} < 10}} \\\left\lbrack {{bdry_{red}^{left}},{bdry}_{red}^{top}} \right\rbrack & {{{for}\mspace{14mu}{\max\left( {W,H} \right)}} = {{8\mspace{14mu}{and}\mspace{14mu}{mode}} \geq 10}} \\\left\lbrack {{bdry_{red}^{top}}\ ,{bdry}_{red}^{left}} \right\rbrack & {{{for}\mspace{14mu}{\max\left( {W,H} \right)}} > {8\mspace{14mu}{and}\mspace{14mu}{mode}} < 6} \\\left\lbrack {{bdry_{red}^{left}},{{bdr}y_{red}^{top}}} \right\rbrack & {{{for}\mspace{14mu}{\max\left( {W,H} \right)}} > {8\mspace{14mu}{and}\mspace{14mu}{mode}} \geq 6}\end{matrix} \right.$

(2) Matrix-Vector Multiplication

In this process, a down-sampled prediction signal pred_(red) of thecurrent block is generated from the reduced boundary vector. pred_(red)is the sum of the matrix-vector product and the offset and may becalculated as follows.pred_(red) =A·bdry_(red) +b

The size of pred_(red) is W_(red)×H_(red). W_(red) and H_(red) aredefined according to the size and shape of the current block as shownbelow. Matrix A has rows as many as W_(red)*H_(red), and has 4 columnswhen W=H=4 or 8 columns in the other cases. The offset vector b is avector of size W_(red)*H_(red).

$W_{red} = \left\{ {{\begin{matrix}4 & {{{for}\mspace{14mu}{\max\left( {W,H} \right)}} \leq 8} \\{\min\left( {W,8} \right)} & {{{for}\mspace{14mu}{\max\left( {W,H} \right)}} > 8}\end{matrix}H_{red}} = \left\{ \begin{matrix}4 & {{{for}\mspace{14mu}{\max\left( {W,H} \right)}} \leq 8} \\{\min\left( {H,8} \right)} & {{{for}\mspace{14mu}{\max\left( {W,H} \right)}} > 8}\end{matrix} \right.} \right.$

Sets S₀, S₁, and S₂ of the matrix A and the offset vector b that may beused for the coding block are predefined for each category of codingblock sizes. The indices (0, 1, 2) of the set S are selected accordingto the aforementioned MipSizeId (i.e., idx(W,H)), and the matrix A andthe offset vector b are extracted from one of the sets S₀, S₁, and S₂according to the MIP mode applied to the current block.

The set S₀ consists of 18 matrices A₀ each having 16 rows and 4 columns,and 18 16-dimensional offset vectors b₀, and is used for a 4×4 block.The set S₁ consists of 10 matrices A₁ each having 16 rows and 8 columnsand 10 16-dimensional offset vectors b₁, and is used for blocks of 4×8,8×4 and 8×8 sizes. Finally, the set S₂ consists of 6 matrices A₂ eachhaving 64 rows and 8 columns and 6 64-dimensional offset vectors b₂, andis used for all other block shapes.

(3) Pixel Interpolation

Interpolation is an upsampling process. As mentioned above, pred_(red)is a down-sampled prediction signal of the original block. In this case,a down-sampled prediction block with a size of predW and predH isdefined as follows.pred_(red) [x][y], with x=0 . . . predW1,y=0 . . . predH−1

A prediction block having an original block size (nTbW, nTbH) generatedby linearly interpolating the prediction signal at the remainingposition in each direction is defined as follows.predSamples[x][y], with x=0 . . . nTbW−1,y=0 . . . nTbH−1

Depending on the horizontal and vertical upsampling scale factorsupHor(=nTbW/predW) and upVer (=nTbH/predH), some or all of thepredSamples are filled from pred_(red) as follows.predSamples[(x+1)*upHor−1][(y+1)*upVer−1]=pred_(red) [x][y]

When upHor=1, all horizontal positions of predSamples from pred_(red)are filled. When upVer=1, all vertical positions of predSamples frompred_(red) are filled.

Thereafter, the remaining empty samples of predSamples are filledthrough bi-linear interpolation. Interpolation in the horizontaldirection and interpolation in the vertical direction are upsamplingprocesses. For interpolation of left and top samples in predSamples,down-sampled samples bdry_(red) ^(top) are assigned to values ofpredSamples[x][−1], and original reference samples on the left areassigned to values of predSamples[−1][y]. The interpolation order isdetermined according to the size of the current block. That is,interpolation is first performed in the direction of the short size.Subsequently, interpolation is performed in the direction of the longsize.

(4) Signaling of MIP Intra-Prediction Mode

For each coding unit (CU) subjected to intra-prediction coding, a flagindicating whether a matrix-based intra-prediction mode (i.e., MIP mode)is applied is transmitted. In VVC draft 5, for signaling the MIP mode,an MPM list consisting of 3 MPMs is used similarly to the traditionalintra-prediction mode (hereinafter, “normal intra-prediction mode”)which is different from the matrix-based intra-prediction. For example,intra_mip_mpm_flag, intra_mip_mpm_idx, and intra_mip_mpm_remainder areused for MIP mode signaling. intra_mip_mpm_idx is coded with a truncatedbinary code, and intra_mip_mpm_remainder is coded with a fixed lengthcode.

Depending on the size of the coding block (CU), up to 35 MIP modes maybe supported. For example, for a CU with max (W, H)<=8 and W*H<32, 35modes are available. In addition, 19 prediction modes and 11 predictionmodes are used for CUs with max(W, H)=8 and max(W, H)>8, respectively.In addition, a pair of modes (two modes) may share a matrix and offsetvector to reduce memory requirements. The specific sharing mode iscalculated as follows. For example, for a 4×4 coding block, mode 19 usesa transposed matrix of the matrix assigned to mode 2.

$m = \left\{ \begin{matrix}{mode} & {{{for}\mspace{14mu} W} = {H = {{4\mspace{14mu}{and}\mspace{14mu}{mode}} < 18}}} \\{{mode} - 17} & {{{for}\mspace{14mu} W} = {H = {{4\mspace{14mu}{and}\mspace{14mu}{mode}} \geq 18}}} \\{mode} & {{{for}\mspace{14mu}{\max\left( {W,H} \right)}} = {{8\mspace{14mu}{and}\mspace{14mu}{mode}} < 10}} \\{{mode} - 9} & {{{for}\mspace{14mu}{\max\left( {W,H} \right)}} = {{8\mspace{14mu}{and}\mspace{14mu}{mode}} \geq 10}} \\{mode} & {{{for}\mspace{14mu}{\max\left( {W,H} \right)}} > {8\mspace{14mu}{and}\mspace{14mu}{mode}} < 6} \\{{mode} - 5} & {{{for}\mspace{14mu}{\max\left( {W,H} \right)}} > {8\mspace{9mu}{and}\mspace{14mu}{mode}} \geq 6}\end{matrix} \right.$

When there is a block to which MIP is applied adjacent to a block towhich a regular intra-prediction mode other than MIP is applied(hereinafter referred to as a “regular block”), a mapping table definedbetween the MIP mode and the regular mode may be used for MPM derivationof the regular block. The mapping table is used to derive a regular modeof similar characteristics from the MIP mode of the neighboring block towhich the MIP is applied. The regular mode derived in this way is usedfor MPM derivation of the regular block. Similarly, even when MIP isapplied to a collocated luma block used in chroma DM derivation, aregular mode of the collocated luma block is derived using the mappingtable and the derived regular mode is used for chroma DM derivation. Theequation below expresses the mapping between the regular modes and theMIP modes using the mapping tables of Tables 3 and 4.predmode_(MIP)=map_regular_to_mip_(idx)[predmode_(regular)]predmode_(regular)=map_mip_to_regular_(idx)[predmode_(MIP)]

TABLE 3 MipSizeId IntraPredModeY[xNbX][yNbX] 0 1 2 0 17 0 5 1 17 0 1 2,3 17 10 3 4, 5 9 10 3 6, 7 9 10 3 8, 9 9 10 3 10, 11 9 10 0 12, 13 17 40 14, 15 17 6 0 16, 17 17 7 4 18, 19 17 7 4 20, 21 17 7 4 22, 23 17 5 524, 25 17 5 1 26, 27 5 0 1 28, 29 5 0 1 30, 31 5 3 1 32, 33 5 3 1 34, 3534 12 6 36, 37 22 12 6 38, 39 22 12 6 40, 41 22 12 6 42, 43 22 14 6 44,45 34 14 10 46, 47 34 14 10 48, 49 34 16 9 50, 51 34 16 9 52, 53 34 16 954, 55 34 15 9 56, 57 34 13 9 58, 59 26 1 8 60, 61 26 1 8 62, 63 26 1 864, 65 26 1 8 66  26 1 8

TABLE 4 MipSizeId IntraPredModeY[xNbX][yNbX] 0 1 2 0 0 0 1 1 18 1 1 2 180 1 3 0 1 1 4 18 0 18 5 0 22 0 6 12 18 1 7 0 18 0 8 18 1 1 9 2 0 50 1018 1 0 11 12 0 12 18 1 13 18 0 14 1 44 15 18 0 16 18 50 17 0 1 18 0 0 1950 20 0 21 50 22 0 23 56 24 0 25 50 26 66 27 50 28 56 29 50 30 50 31 132 50 33 50 34 50

As described above, in VVC draft 5, when a block is predicted based onMIP, reconstructed neighboring samples on the left side of the block andreconstructed neighboring samples on the above of the block are alwaysused as reference samples. This approach may deteriorate predictionperformance in the case of a block in which the texture of the block hasdirectional characteristics. In addition, in VVC draft 5, forinterpolation of top samples in predSamples, the down-sampled sample setis allocated to values of predSamples[x][−1] as shown in FIG. 5 ,thereby making the interpolation process more complex than necessary.Furthermore, in VVC draft 5, as each MPM list is employed for signalingof the MIP mode and the regular mode, the implementation may be verycomplicated due to many checks and conditions such as a requirement formapping between the MIP mode and the regular mode.

In view of the above, the present disclosure proposes several improvedtechniques capable of reducing the implementation complexity of the MIPmode and improving prediction performance.

Use of Smoothing Filtered Reference Sample

In typical intra-prediction coding, a smoothing filter, a Gaussianinterpolation filter, and a cubic interpolation filter may beselectively used to obtain reference samples for directional modes. FIG.6 illustrates smoothing filtering and interpolation filtering. In thereference sample filtering process, intra-prediction modes areclassified into three groups. Group A is composed of horizontal andvertical prediction modes, Group B is composed of diagonal modes thatare multiples of 45 degrees, and Group C is composed of otherdirectional modes. No filter is applied for group A. The diagonal modesbelonging to group B refer to a pixel at an integer position, andtherefore there is no need to apply the interpolation filter, and only a[1, 2, 1]/4 smoothing filter is applied to the reference samples. Forgroup C, the [1, 2, 1]/4 smoothing filter is not applied, but a 4-tapGaussian interpolation filter or a 4-tap cubic interpolation filter isapplied according to conditions to obtain reference samples at decimalpositions.

Smoothing filtering for reference samples may be performed as follows.Hereinafter, a reference sample prior to filtering is indicated asrefUnfilt[x][y], and a reference sample after filtering is indicated asrefFilt[x][y]. Here, the [1, 2, 1]/4 smoothing filter is used forfiltering. refH and refW are the number of left reference samples andthe number of above reference samples, respectively.

-   -   Filtering of the above left corner reference sample        refFilt[−1][−1]=(refUnfilt[−1][0]+2*refUnfilt[−1][−1]+refUnfilt[0][−1]+2)>>2    -   Filtering of the left reference sample        refFilt[−1][y]=refUnfilt[−1][y+1]+2*refUnfilt[−1][y]+refUnfilt[−1][y−1]+2)>>2        {for y=0 . . . refH−2}        refFilt[−1][refH−1]=refUnfilt[−1][refH−1]    -   Filtering of the above reference sample        refFilt[x][−1]=(refUnfilt[x−1][−1]+2*refUnfilt[x][−1]+refUnfilt[x+1][−1]+2)>>2        {for x=0 . . . refW−2}        refFilt[refW−1][−1]=refUnfilt[refW−1][−1]

As shown in the equations above, filtering may be performed betweenadjacent reference samples located on the same reference line. However,filtering may also be performed between reference samples located ondifferent reference lines. For example, a filtered reference sample maybe obtained by calculating the average of two samples located at MRLindexes 0 and 1.

As described above, when MIP is used, boundary samples of an adjacentdecoded block located at the above and the left of the coding block mayconstitute reference samples. In this case, the boundary samples onwhich no filtering has been performed are used as reference samples.Considering that the MIP mode is often used for predicting andreconstructing low frequency components, it may be advantageous to usesample values from which high frequency components are removed byfiltering the reference samples in the MIP prediction process.

In accordance with an aspect of the present disclosure, as in thedirectional modes of regular intra-prediction, reference samples may befiltered and then used in the MIP mode. The filtered reference samplesand unfiltered reference samples may be adaptively used according to theprediction mode of a block, the size of the block, and the MIP modetype.

Filtering for reference samples may be performed when all the followingconditions are true. Therefore, if any of the following conditions isnot satisfied, the values of refUnfil are copied to refFilt withoutfiltering. In some cases, when the Intra Subpartition (ISP) is appliedto the coding block, if the width or height of the partitioned subblockis greater than or equal to 16, filtering may be performed on theboundary of the corresponding sides.

-   -   The MRL index is 0.    -   The product of nTbW and nTbH is greater than 32.    -   CIdx is 0 (i.e., it is a luma sample).    -   IntraSubPartitionsSplitType is ISP_NO_SPLIT (i.e., ISP is not        applied).    -   RefFilterFlag is 1 (i.e., the intra-prediction mode is one of 0,        14, 12, 10, 6, 2, 34, 66, 72, 76, 78, 80, or one of the MIP        modes).

Whether to adaptively perform filtering of reference pixels may bedetermined according to the size of a block to which MIP is applied. Asdescribed above, the sizes of blocks to which MIP is applied areclassified into three categories, and whether to filter the referencepixel may be determined depending on idx(W,H). For example, refUnfiltmay be used when Idx is less than 2 (i.e., blocks having a size of 4×4,4×8, or 8×4), and refFilt may be used when Idx is 2, or vice versa. Asanother example, refUnfilt is used when Idx is less than 1, and refFiltis used when Idx is greater than or equal to 1, or vice versa. Asanother example, refUnFilt may be used when the size of the block is4×4, 4×8, or 8×4, and refFilt may be used otherwise, or vice versa. Asanother example, refFilt may not be used when either the width or theheight of the block is equal to 4. As another example, refFilt may notbe used when any one of the width and height of the block is equal to 4.

Whether to adaptively perform filtering of the reference pixel may bedetermined according to the shape of a block to which MIP is applied.For example, refFilt may be used only when the width and height of thecoding block are the same. As another example, refFilt may be used onlywhen the width and height of the coding block are different. As anotherexample, the width and height of a coding block may be compared, refFiltmay be used only for the boundary of a block for a longer side.Alternatively, refFilt may be used only for the boundary of a blockwhose length is short. As another example, refFilt may be used only forthe boundary of a side that is greater than or equal to 16 (or 32) ofthe width or height of a block.

In some embodiments, reference sample sets bdry^(left) and bdry^(top)around a MIP-coded block may be configured as follows. For example, thereference sample sets may be configured from luma samples mapped afterLMCS (Luma Mapping with Chroma Scaling) around the MIP-coded block, ormay be configured from luma samples mapped before LMCS around theMIP-coded block. As another example, the reference sample sets may beconfigured from luma samples of neighboring blocks before performingblock boundary filtering for regular intra-prediction, or may beconfigured from luma samples of the neighboring blocks after blockboundary filtering. As another example, the reference sample sets may beconfigured through the same process as used to perform regularintra-prediction around the MIP-coded block.

Generation of an Input Boundary Vector

As described above, in predicting a coding block based on MIP,reconstructed neighboring samples bdry^(left) at the left of the codingblock and reconstructed neighboring samples bdry^(top) at the above ofthe coding block may be used. However, this method may deteriorateprediction performance in the case of a coding block whose texture hasdirectional characteristic.

Accordingly, it may be advantageous to selectively use a neighboringsample set to be used to generate an input boundary vector betweenbdry^(left) and bdry^(top) so as to reflect the directionalcharacteristic that the texture of the block may have. For example, whenpixels of a current coding block are horizontal characteristic, aprediction signal may be generated using left neighboring samples.Similarly, when pixels of the current coding block are verticalcharacteristic, a prediction signal may be generated using aboveneighboring samples. The neighboring sample set to be used to generatethe input boundary vector may be determined differently according to theMIP mode applied to the coding block.

Hereinafter, for simplicity, several methods of determining a boundaryvector and determining predicted samples from the boundary vector willbe described for a case of using left neighboring samples and a case ofusing above neighboring samples.

A. Constructing a Boundary Vector Using Left Neighboring Samples

FIGS. 7A to 7C are conceptual diagrams illustrating an exemplary methodof constructing a boundary vector to be input to matrix-vectormultiplication, using left neighboring samples.

As an example, as illustrated in FIG. 7A, when the size (height) of thecurrent coding block is the same as the size of the boundary vectorbdry_(red), the boundary vector bdry_(red) of the same size may befilled using the left neighboring sample set bdry^(left). For example,each of the left neighboring samples may be included in the entry of theboundary vector.

As another example, as illustrated in FIG. 7B, the boundary vectorbdry_(red) may be filled using a down-sampled sample set bdry_(red)^(left) obtained from the left neighboring sample set bdry^(left). Forexample, bdry_(red) ^(left) may be obtained by averaging bdry^(left) forevery two samples.

As still another example, as illustrated in FIG. 7C, the boundary vectorbdry_(red) may be filled by calculating an average of two pixels of eachrow using two left columns adjacent to the coding block.

Depending on the size of the coding block, bdry^(left) and bdry_(red)^(left) may be used adaptively. For example, as shown in the equationbelow, when H<=8, bdry^(left) is used. Otherwise, bdry_(red) ^(left) maybe used.

$\begin{matrix}{{b{dry}_{red}} = \left\{ \begin{matrix}\left\lbrack {bdry}^{left} \right\rbrack & {{{for}\mspace{14mu} H}<=8} \\\left\lbrack {bdry}_{red}^{left} \right\rbrack & {otherwise}\end{matrix} \right.} & \;\end{matrix}$

In some embodiments, in generating a down-sampled sample set bdry_(red)^(left) from bdry^(left), down-sampling may be performed in a differentmanner according to characteristics of video content. As in screencontent coding (SCC), video content may have the same pixel values orintensity values in a specific region, or may have pixel values orintensity values that gradually increases/decreases. In this case, itmay be useful to reduce downsampling complexity to generate thedown-sampled sample set bdry_(red) ^(left) by sampling only the entriescorresponding to the even positions (or odd positions) of bdry^(left),rather than generating the down-sampled sample set bdry_(red) ^(left)from bdry^(left) by the averaging operation.

For example, as shown in the equation below, when bdry^(left) has Hentries from [0] to [H−1] and bdry_(red) ^(left) having the size of H/2is generated therefrom, bdry^(left) may be used if H≤8. Otherwise,bdry_(red) ^(left) may be generated by taking entries corresponding toeven positions including 0.

$\begin{matrix}{{bdr{y_{red}^{left}(i)}} = \left\{ \begin{matrix}\left\lbrack {{bdry}^{left}(i)} \right\rbrack & {{{for}\mspace{14mu} H}<=8} \\\left\lbrack {bdr{y^{left}\left( {i*2} \right)}} \right\rbrack & {otherwise}\end{matrix} \right.} & \;\end{matrix}$

Furthermore, downsampling complexity may be further lowered by fillingall entries of bdry_(red) ^(left) with the entry value at the firstposition in bdry^(left).bdry_(red) ^(left)(i)=bdry^(left)(0)

These simplified downsamplings and the downsampling through theaveraging operation may be selected depending on the characteristics ofthe video content. For example, when intra-block copy (IBC), which is acoding tool suitable for screen content coding, is applied to a codingblock positioned on the left of the MIP-coded current block, simplifieddownsampling taking entries corresponding to even-numbered positions (orodd-numbered positions) may be used for the coding block. As anotherexample, when a transform skip mode, which is frequently used in screencontent coding, is applied to a coding block positioned on the left ofthe MIP-coded current block, simple downsampling taking entriescorresponding to even-numbered positions (or odd-numbered positions) maybe used for the coding block. As another example, downsampling throughthe averaging operation may be completely replaced by simplifieddownsampling taking entries corresponding to even positions (or oddpositions).

B. Boundary Vector Construction Using Above Neighboring Samples

FIGS. 8A to 8C are conceptual diagrams illustrating an exemplary methodof constructing a boundary vector to be input to a matrix-vectormultiplication operation, using above neighboring samples.

As an example, as illustrated in FIG. 8A, when the size (width) of thecurrent coding block is the same as the size of the boundary vectorbdry_(red) bdry_(red), the boundary vector bdry_(red) of the same sizemay be filled using a above neighboring sample set. For example, each ofthe above neighboring samples may be included in the entry of theboundary vector.

As another example, as illustrated in FIG. 8B, the boundary vectorbdry_(red) may be filled using a down-sampled sample set bdry_(red)^(top) obtained from the above neighboring sample set bdry^(top). Forexample, bdry_(red) ^(top) may be obtained by averaging bdry^(top) forevery two samples.

As another example, as illustrated in FIG. 8C, the boundary vectorbdry_(red) may be filled by calculating an average of two pixels in eachcolumn using two above rows adjacent to the coding block.

Depending on the size of the coding block, bdry^(top) and bdry_(red)^(top) may be used adaptively. For example, as shown in the equationbelow, when W<=8, bdry^(top) may be used. Otherwise, bdry_(red) ^(top)may be used.

$\begin{matrix}{{b{dry}_{red}} = \left\{ \begin{matrix}\left\lbrack {bdry}^{top} \right\rbrack & {{{for}\mspace{14mu} W}<=8} \\\left\lbrack {bdry}_{red}^{top} \right\rbrack & {otherwise}\end{matrix} \right.} & \;\end{matrix}$

As described above, it is useful in reducing downsampling complexity togenerate the down-sampled sample set bdry_(red) ^(top) by sampling onlythe entries corresponding to the even positions (or odd positions) ofbdry^(top), rather than generating the down-sampled sample setbdry_(red) ^(top) from bdry^(top) through the averaging operation.

For example, as shown in the equation below, when bdry^(top) has Wentries from [0] to [W−1], and bdry_(red) ^(top) having a size of W/2 isgenerated therefrom, bdry^(top) may be used if W≤8. Otherwise,bdry_(red) ^(top) may be generated by taking entries corresponding toeven positions including 0.

$\begin{matrix}{{bdr{y_{red}^{top}(i)}} = \left\{ \begin{matrix}\left\lbrack {{bdry}^{top}(i)} \right\rbrack & {{{for}\mspace{14mu} W}<=8} \\\left\lbrack {bdr{y^{top}\left( {i*2} \right)}} \right\rbrack & {otherwise}\end{matrix} \right.} & \;\end{matrix}$

Furthermore, downsampling complexity may be further lowered by fillingall entries of bdry_(red) ^(top) with the entry value at the firstposition in bdry^(top).bdry_(red) ^(top)(i)=bdry^(top)(0)

These simplified downsamplings and the downsampling through theaveraging operation may be selected depending on the characteristics ofthe video content. For example, when intra-block copy (IBC), which is acoding tool suitable for screen content coding, is applied to a codingblock positioned on the left (or above) of the MIP-coded current block,simplified downsampling taking entries corresponding to even-numberedpositions (or odd-numbered positions) may be used for the coding block.As another example, when a transform skip mode, which is frequently usedin screen content coding, is applied to a coding block positioned on theleft (or above) of the MIP-coded current block, simplified downsamplingtaking entries corresponding to even-numbered positions (or odd-numberedpositions) may be used for the coding block. As another example,downsampling through the averaging operation may be completely replacedby simple downsampling taking entries corresponding to even-numberedpositions (or odd-numbered positions).

Matrix-Vector Multiplication

According to the MIP technique described in VVC draft 5, a boundaryvector bdry_(red) of length 4 or 8 is obtained from the left neighboringsample set and the above neighboring sample set. The boundary vectorbdry_(red) is input to the vector-matrix multiplication operation.Rather than applying the vector-matrix multiplication directly to theboundary vector bdry_(red), it may be more advantageous in terms ofcomputational and hardware complexity to remove the DC component fromthe boundary vector bdry_(red) prior to applying the vector-matrixmultiplication, and add a DC component after applying the vector-matrixmultiplication. According to this method, all entries of a weight matrixused for the vector-matrix multiplication may be expressed as unsignedintegers. That is, it may be advantageous to make the average of theentries included in the boundary vector bdry_(red) zero or to convertthe same to a value close to zero before applying the vector-matrixmultiplication to the boundary vector bdry_(red).

As an example, before applying the vector-matrix multiplication, onepixel value belonging to bdry_(red) may be subtracted from each entry ofbdry_(red). As another example, before applying the vector-matrixmultiplication, the average of bdry_(red) may be subtracted from eachentry of bdry_(red). As still another example, before applying thevector-matrix multiplication, a pixel value of the first entry ofbdry_(red) may be subtracted from each entry of bdry_(red).

As an exemplary implementation, the video encoder and decoder maycalculate an average of the boundary vector bdry_(red) and apply thevector-matrix multiplication to a vector obtained by subtracting theaverage from each entry of the boundary vector bdry_(red). An inputboundary vector input_(red) to be input to the vector-matrixmultiplication operation may be defined as follows. Here, p_avr is theaverage of the boundary vector bdry_(red), and bitDepth denotes the lumabit-depth.

When MipSizeId (=idx(W,H)) is 0 or 1,input_(red)[0]=p_avr−(1<<(bitDepth−1))input_(red) [j]=bdry_(red) [j]−p_avr,j=1, . . . ,size(bdry_(red))−1.

When MipSizeId (=idx(W,H)) is 2,input_(red) [j]=bdry_(red) [j+1]−p_avr,j=0, . . . ,size(bdry_(red))−2

As another exemplary implementation, in order to avoid calculating theaverage, the average may be replaced with the first entry of theboundary vector bdry_(red). In this case, the input boundary vectorinput_(red) to be input to the vector-matrix multiplication operationmay be defined as follows.

When MipSizeId (=idx(W,H)) is 0 or 1,input_(red) [j]=bdry_(red) [j]−bdry_(red)[0],j=1, . . .,size(bdry_(red))−1input_(red)[0]=bdry_(red)[0]−(1<<(bitDepth−1))

When MipSizeId (=idx(W,H)) is 2,input_(red) [j]=bdry_(red) [j+1]−bdry_(red)[0],j=0, . . .,size(bdry_(red))−2.

That is, when MipSizeId is 0 or 1, the first entry of the input boundaryvector input_(red) is obtained based on the difference between half(‘1<<(bitDepth−1)’) of the maximum value that can be expressed in bitdepth and the first entry of the boundary vector bdry_(red), andsubsequent entries of the input boundary vector input_(red) are obtainedbased on subtraction of the value of the first entry from each entry ofthe boundary vector bdry_(red). When MipSizeId=2, the differentialvector input_(red) has a length of 7, and accordingly the weightmatrices A2 of the set S2 used for this case each have 64 rows and 7columns (in VVC draft 5, the weight matrices A2 each have 64 rows and 8columns).

In addition, by using the matrix mWeight[x][y] obtained bypre-subtracting the offset vector from the weight matrices (A0, A1, A2),the offset vector b may be removed from pred_(red)=A·bdry_(red)+b, butpredicted values with a slight error is obtained. According to theimproved method, the predicted sample set pred_(red)[x][y] may becalculated as follows.pred_(red) [x][y]=(((Σ_(i=0) ^(inSize-1)mWeight[i][y*predSize+x]*input_(red) [i])+oW)>>6)+bdry_(red)[0]

Here, oW=32−32*(Σ_(i=0) ^(inSize-1)p[i]), and inSize is the size ofinput_(red)[j].

Linear Interpolation

Interpolation is required when the number of entries in pred_(red) issmaller than the number of samples in the prediction block. Adown-sampled prediction block with a size of predW and predH is definedas follows.pred_(red) [x][y], with x=0 . . . predW−1,y=0 . . . predH−1

A prediction block having a size (nTbW, nTbH) corresponding to a codingblock, in which the prediction signals at the remaining positions wouldbe generated by linear interpolation in each direction, is defined asfollows.predSamples[x][y], with x=0 . . . nTbW−1,y=0 . . . nTbH−1

Depending on the horizontal and vertical upsampling scale factors upHor(=nTbW/predW) and upVer (=nTbH/predH), some or all of the predSamplesare filled from pred_(red) as follows.predSamples[(x+1)*upHor−1][(y+1)*upVer−1]=pred_(red) [x][y]

When upHor=1, all positions in the horizontal direction of predSamplesare filled from pred_(red). When upVer=1, all positions in the verticaldirection of predSamples are filled from pred_(red).

Thereafter, the remaining empty samples of predSamples are filledthrough bi-linear interpolation. Interpolation in the horizontaldirection and interpolation in the vertical direction are upsamplingprocesses. The interpolations may be performed in a fixed orderregardless of the size of the coding block. For example, interpolationmay be performed first in the horizontal direction of the coding block,followed by interpolation in the vertical direction. In this case,clipping may be performed before upsampling, such that values ofpred_(red)[x][y] or predSamples[x][y] are between 0 and 2^(bitDepth)−1.2^(bitDepth)−1 is the maximum value that can be expressed in bit depth.

For interpolation, pred_(red) and reference pixels of a neighboringblock may be referred to as follows. For example, above originalreference samples may be allocated to values of predSamples[x][−1], andleft original reference samples may be allocated to values ofpredSamples[−1][y]. As another example, luma samples in neighboringblocks before or after LMCS may be allocated to predSamples[x][−1]positions and predSamples[−1][y] positions. As still another example,luma samples in neighboring blocks before or after block boundaryfiltering for intra-prediction around a coding block may be allocated topredSamples[x][−1] positions and predSamples[−1][y] positions.

mip_transpose_flag

As described above, VVC draft 5 supports up to 35 MIP modes according tothe size and shape of a CU. For example, 35 modes are available for a CUwith max(W, H)<=8 && W*H<32. For CUs each having max(W, H)=8 and max(W,H)>8, 19 and 11 modes are used, respectively. In addition, a pair ofmodes (two modes) may share a matrix and offset vector to reduce memoryrequirements. For example, for a 4×4 coding block, mode 19 uses atransposed matrix of the matrix assigned to mode 2. Furthermore, byconcatenating bdry_(red) ^(top) and bdry_(red) ^(left) in an orderdetermined according to the MIP mode and the size of the block, aboundary vector bdry_(red) to be input to the vector-matrixmultiplication is generated.

An improved approach may be used that may achieve substantiallyequivalent levels of coding efficiency while reducing complexity.According to another aspect of the present disclosure, instead of theexisting method in which one mode uses the transpose of the matrix usedby the other mode, a new method configured to change the order ofconcatenating bdry_(red) ^(top) and bdry_(red) ^(left) constituting aboundary vector bdry_(red) used for vector-matrix multiplication foreach mode may be used. The video encoder may signal a syntax element(mip_transpose_flag) indicating an order in which bdry_(red) ^(top) andbdry_(red) ^(left) are concatenated to constitute the boundary vectorbdry_(red) for each mode. When the syntax element (mip_transpose_flag)indicates concatenation in order of bdry_(red) ^(left) and bdry_(red)^(top), a prediction matrix obtained through the matrix-vectormultiplication operation may also be transposed. According to thismethod, the number of available MIP modes may be reduced by halfcompared to the conventional method, and coding complexity in terms ofboundary vector generation and vector-matrix multiplication may bereduced.

Signaling of MIP Mode

For a coding unit (CU) coded in the intra-prediction mode, a flagindicating whether the intra-prediction type is matrix-basedintra-prediction (MIP) may be signaled. When MIP is applied to a currentCU, a syntax element which indicates a MIP mode used in the current CUamong a plurality of available MIP modes may be additionally signaled.

Unlike the traditional intra-prediction mode as shown in FIGS. 3A and 3B(i.e., regular intra-prediction mode), the MPM list may not be used forsignaling the MIP mode. Instead, for example, one syntax element (e.g.,intra_mip_mode) that indicates an MIP mode used in the current CU amongthe plurality of MIP modes and may be coded with a truncated binary codemay be used.

A part of an exemplary coding unit syntax proposed based on the VVCdraft 5 is provided below. In the syntax below, the graying of elementsis used to provide understanding.

TABLE 5 if( CuPredMode[ x0 ][ y0 ] = = MODE_INTRA ) {  if( treeType = =SINGLE_TREE | | treeType = =  DUAL_TREE_LUMA ) { if(sps_bdpcm_enabled_flag &&  cbWidth <= MaxTsSize && cbHeight <=MaxTsSize )  intra_bdpcm_flag[ x0 ][ y0 ] if( intra_bdpcm_flag[ x0 ][ y0] )  intra_bdpcm_dir_flag[ x0 ][ y0 ] else { if( sps_mip_enabled_flag && ( Abs( Log2( cbWidth ) − Log2( cbHeight ) )<= 2 ) &&  cbWidth <= MaxTbSizeY && cbHeight <=  MaxTbSizeY )intra_mip_flag[ x0 ][ y0 ]  if( intra_mip_flag[ x0 ][ y0 ] )intra_mip_mode[ x0 ][ y0 ]

When intra_mip_flag[x0][y0] is 1, it indicates that the intra-predictiontype of the current block is MIP. When intra_mip_flag[x0][y0] is 0, itindicates that the intra-prediction type of the current block is regularintra-prediction, not MIP. When intra_mip_flag[x0][y0] is not present,it may be inferred to be equal to 0. intra_mip_mode[x0][y0] indicates anMIP mode used for the current block in MIP, and is expressed as atruncated binary code.

MPM (Most Probable Mode)

In a conventional approach, intra-prediction coding employing MostProbable Mode (MPM) may be used. For example, in HEVC, a list of threeMPMs is configured from the intra-prediction modes of the left and aboveblocks. The drawback of this method is that more modes (intra-modesother than the MPM) belong to non-MPMs that need to be coded with morebits. Several methods have been proposed to extend the number of MPMs to3 or more entries (e.g., 6 MPM modes). However, configuring such an MPMlist with more entries may require more checks and conditions, which maymake implementation more complex.

In order to keep the complexity of configuration of an MPM list low, anMPM list including six MPM candidates may be configured usingintra-prediction modes of a left neighboring block and an aboveneighboring block adjacent to the current block. The MPM candidates mayinclude a default intra-prediction mode (e.g., a PLANAR mode), anintra-prediction mode of a neighboring block, and an intra-predictionmode derived from the intra-prediction mode of the neighboring block.When the intra-prediction mode of the neighboring block is not used (forexample, when the neighboring block is inter-predicted, or theneighboring block is located in a different slice or another tile), theintra-prediction mode of the neighboring block may be set to PLANAR.

According to the type of intra-prediction mode of the mode (Left) of theleft block and the mode (Above) of the above block, it is largelydivided into 4 cases. When Left and Above are different from each other,and the two modes are directional modes, it may be further dividedaccording to the difference of the Left and Above to generate an MPMlist. In the table below, Max refers to the larger mode between the Leftand the Above, and MIN refers to the smaller mode between the Left andthe Above.

TABLE 6 Condition Detailed condition MPM modes Left mode and Above modeare directional mode {Planar, Left, Left − 1, Left + 1, Left − 2, Left +2} and are the same Left mode and Above mode 2 ≤ Max-Min ≤ 62 {Planar,Left, Above, DC, Max − 1, Max + 1} are different, both modes otherwise{Planar, Left, Above, DC, Max − 2, Max + 2} are directional mode Leftmode and Above mode are different, and {Planar, Max, Max − 1, Max + 1,Max − 2, Max + 2} only one of them is directional mode Left mode andAbove mode are non-directional {Planar, DC, Angular50, Angular18,Angular46, mode (i.e., Planar or DC) Angular54}

The video encoder may signal a 1-bit flag (e.g., mpm_flag) indicatingwhether the intra-prediction mode of the current block corresponds toMPM. Typically, when the intra-prediction mode of the current blockcorresponds to MPM, an MPM index indicating one of 6 MPMs isadditionally signaled. Note that in Table 6, the PLANAR mode is alwaysincluded in the MPM list. That is, 6 MPMs may be divided into PLANAR and5 non-PLANAR MPMs. Therefore, it may be efficient that the encoderexplicitly signals whether the intra-prediction mode of the currentblock is the PLANAR mode (e.g., using a 1-bit flag) when theintra-prediction mode of the current block is an MPM mode, andadditionally signals an MPM index indicating one of the other fivenon-PLANAR MPMs when the intra-prediction mode of the current block isthe same as one of the other five non-PLANAR MPMs. When theintra-prediction mode of the current block does not correspond to anyMPM, a syntax element indicating one of the remaining 61 non-MPMsexcluding the 6 MPMs may be encoded using a truncated binary code.

A. Removal of a Mapping Table Between MIP Mode and Regular Mode

In VVC draft 5, an MPM list is used for signaling of the MIP mode andthe regular mode, and a mapping table between the MIP mode and theregular mode is required to configure the MIP list. Due to thecharacteristics of the MIP technique including an averaging operationand an interpolation operation, the residual signal of a block to whichMIP is applied may have a low frequency component dominant in thetransform domain. The characteristics of the residual signal may besimilar to the residual signal of a block to which the PLANAR mode or DCmode is applied. Therefore, in deriving an MPM list of blocks coded inthe regular intra-prediction mode, the use of a mapping table betweenthe MIP mode and the regular mode may be avoided from the perspective ofthe similarity of the residual signals.

According to an aspect of the present disclosure, in deriving an MPMlist for a block (i.e., a regular block) coded in the regularintra-prediction mode, when a neighboring block is coded in the MIPmode, the intra-prediction mode of the neighboring block may be regardedas the PLANAR mode (or DC mode). For example, when the MIP mode isapplied to the neighboring block, the PLANAR mode (or DC mode) may beadded to the MPM list instead of the MIP mode of the neighboring block.Thereby, the need for the encoder and decoder to store the mapping tablebetween the MIP mode and the regular mode in the memory is eliminated.

Similarly, even when a chroma DM (direct mode) is derived, if MIP isapplied to a collocated luma block, the intra-prediction mode of theluma block may be regarded as the PLANAR mode (or DC mode), instead ofusing a mapping table between the MIP mode and the regular mode. Thevideo decoder parses a syntax element specifying an intra-predictionmode for a chroma block, and the syntax element may indicates that theintra-prediction mode of the chroma block employs the intra-predictionmode of the collocated luma block. In such case, when MIP is applied tothe collocated luma block, the intra-prediction mode of the luma blockmay be regarded as the PLANAR mode (or DC mode). That is, when MIP isapplied to a collocated luma block in the chroma direct mode (DM), itmay be determined that the intra-prediction mode of the chroma block isthe PLANAR mode (or DC mode).

B. Integration of MIP Mode into Regular Intra-Mode

In VVC draft 5, MIP is treated as a separate intra-prediction typedifferent from regular intra-prediction, and whether MIP is used issignaled at the CU level, using intra_mip_flag. When MIP is used, an MIPmode selected for the current CU is coded. When the MIP is not used, theregular intra-prediction mode selected for the current CU is coded. Inthe following, an alternative approach that may improve the signalingefficiency of the intra-prediction mode is presented. The proposedapproach is based on replacing one of the regular modes (e.g. PLANAR,DC, or directional mode) with an MIP mode or adding the MIP mode as oneof the regular modes. Replacing one of the regular modes with the MIPmode may be useful when there is possible redundancy between the MIPmode and a certain regular mode. In this case, the proposed approach mayimprove the signaling efficiency of the intra-prediction mode.

A description will be exemplarily given of a method of decoding theintra-prediction mode of a coding block in the case where the DC modeamong the 67 modes described above is replaced with the MIP mode.

The video decoder may use the MPM list to determineintra_predictiton_mode[x0][y0]] for the coding block. The video decodermay decode MPM related syntax elements to determineintra_predictiton_mode[x0][y0]. When the value ofintra_predictiton_mode[x0][y0] is one of {0, 2, . . . , 66}, theintra-prediction type of the coding block is set to regularintra-prediction. When the value of intra_predictiton_mode[x0][y0] is 1(which was originally the mode index of INTRA DC), the video decodersets the intra-prediction type of the coding block to MIP, and decodes asyntax element indicating an MIP prediction mode used for encoding ofthe coding block.

An MPM list including, for example, 6 MPM candidates may be configuredusing intra-prediction modes of a left block and an above block adjacentto the current coding block. It is largely divided into 4 casesdepending on whether the mode (Left) of the left block and the mode(Above) of the above block are directional modes. When Left and Aboveare different from each other, and both modes are directional modes, twomore cases may be included according to the difference of the Left andAbove to generate an MPM list. In Table 7 below, Max refers to thelarger mode between the Left and the Above, and MIN refers to thesmaller mode between the Left and the Above.

TABLE 7 Condition Detailed condition MPM modes Left mode and Above modeare directional mode {Planar, Left, Left − 1, Left + 1, MIP, Left − 2}and are the same Left mode and Above mode 2 ≤ Max-Min ≤ 62 {Planar,Left, Above, MIP, Max − 1, Max + 1} are different, both modes otherwise{Planar, Left, Above, MIP, Max − 2, Max + 2} are directional mode Leftmode and Above mode are different, and {Planar, Max, MIP, Max − 1, Max +1, Max − 2} only one of them is directional mode Left mode and Abovemode are non-directional {Planar, MIP, Angular50, Angular18, Angular46,mode (i.e., Planar or DC) Angular54}

Note that in Table 7, the MIP mode is always included in the MPM list.That is, six MPMs may be divided into one MIP and five non-MIP MPMs.Therefore, it may be efficient for the encoder 1) to first signalwhether the intra-prediction mode of the current block is an MIP mode(e.g., using a 1-bit flag) when the intra-prediction mode of the currentblock is an MPM mode and 2) to additionally signal an MPM indexindicating one of the other five non-MIP MPMs when the intra-predictionmode of the current block is not the MIP mode. When the intra-predictionmode of the current block is not any MPM, a syntax element indicatingone of the remaining 61 non-MPMs excluding the six MPMs may be encodedusing a truncated binary code.

While the regular intra-prediction mode may be applied together with theMultiple reference line (MRL) and Intra sub-partition (ISP) of VVC, theMIP mode is available only in the case where the MRL index is 0 (thatis, reference samples of the first line are used) and in the case wherethe ISP is not applied. However, the MIP mode may be availableregardless of whether MRL and ISP are applied.

A part of an exemplary intra-prediction mode related syntax is providedbelow. In the syntax below, the graying of elements is used to provideunderstanding.

TABLE 8  if( intra_luma_ref_idx[ x0 ][ y0 ] = = 0 ) intra_luma_mpm_flag[x0 ][ y0 ]  if( intra_luma_mpm_flag[ x0 ][ y0 ] ) { if(intra_luma_ref_idx[ x0 ][ y0 ] = = 0 )  intra_luma_not_MIP_flag[ x0 ][y0 ]  if( !intra_luma_not_MIP_flag[ x0 ][ y0 ] ) intra_mip_mode[ x0 ][y0 ] if( intra_luma_not_MIP_flag[ x0 ][ y0 ] )  intra_luma_mpm_idx[ x0][ y0 ]  } else intra_luma_mpm_remainder[ x0 ][ y0 ] }

The intra-prediction mode for the coding block of the luma component maybe signaled using syntax elements including intra_luma_mpm_flag,intra_luma_not_MIP_flag, intra_luma_mpm_idx, intra_mip_mode, andintra_luma_mpm_remainder.

intra_luma_mpm_flag indicates whether the intra-prediction mode of thecoding block is an MPM mode. When intra_luma_mpm_flag is not present, itis inferred that intra_luma_mpm_flag is equal to 1.intra_luma_not_MIP_flag indicates whether the intra-prediction mode ofthe coding block is the MIP mode. When intra_luma_not_MIP_flag is 1, itindicates that the intra-prediction mode of the coding block is not theMIP mode (that is, it is a regular intra-mode). intra_mip_mode mayspecify one MIP mode among a plurality of MIP modes available for thesize of the current coding block, and may be coded with a truncatedbinary code. When intra_luma_mpm_flag is not present, it is inferredthat intra_luma_mpm_flag is equal to 1.

intra_luma_mpm_idx specifies one MPM mode identical to theintra-prediction mode of the coding block among the five non-MIP MPMs.intra_luma_mpm_remainder specifies one non-MPM mode identical to theintra-prediction mode of the coding block among the non-MPMs.intra_luma_mpm_remainder may be coded with a truncated binary code.

Referring to Table 8, when intra_luma_mpm_flag is 1 and MRL INDEX is 0(i.e., intra_luma_ref_idx=0), the video decoder parsesintra_luma_not_MIP_flag. When intra_luma_not_MIP_flag is 0, theintra-prediction type of the coding block is MIP, and thus the videodecoder decodes intra_mip_mode to identify the MIP mode used in thecoding block. When intra_luma_not_MIP_flag is 1, the video decoderconfigures an MPM list consisting of 5 non-MIP MPMs and decodesintra_luma_mpm_idx indicating MPM INDEX. When intra_luma_mpm_flag is 0,intra_luma_mpm_remainder is decoded.

FIG. 9 is a flowchart illustrating a method of decoding video dataadopting some of the above improvements according to an embodiment ofthe present disclosure.

The video decoder may decode a syntax element indicating anintra-prediction type of the current block of video data from thebitstream (S910). Intra-prediction types include matrix basedintra-prediction (MIP) and regular intra-prediction. The syntax elementmay be a truncated binary code specifying one of a plurality of MIPmodes allowed for the size and shape of the current block.

The video decoder may generate a prediction block for the current blockby selectively performing MIP or regular intra-prediction based on theintra-prediction type of the current block.

In generating the prediction block for the current block by performingregular intra-prediction, the video decoder may perform the followingoperations (S920 to S940). The video decoder may configure an MPM listfor the current block by deriving Most Probable Mode (MPM) candidatesbased on the regular intra-prediction mode of the neighboring blocksadjacent to the current block (S920), and derive a regularintra-prediction mode for the current block based on the MPM list(S930). In deriving MPM candidates based on the regular intra-predictionmode of the neighboring blocks, the video decoder may set (regard) theregular intra-prediction mode of the neighboring blocks as the PLANARmode when the intra-prediction type of the neighboring blocks ismatrix-based intra-prediction. The decoder may generate a predictionblock for the current block based on the regular intra-prediction modeof the current block (S940).

In generating the prediction block for the current block by performingmatrix-based intra-prediction, the video decoder may perform thefollowing operations (S921 to S951). The video decoder may decode asyntax element indicating a matrix-based intra-prediction mode for thecurrent block from the bitstream in order to determine the matrix-basedintra-prediction mode for the current block (S921). The video decodermay derive an input boundary vector using neighboring samples adjacentto the current block based on the width and height of the current block(S931), and may generate predicted samples for the current block basedon matrix-vector multiplication between a matrix predefined for thematrix-based intra-prediction mode for the current block and the inputboundary vector (S941). The video decoder may derive the predictionblock for the current block by performing clipping and linearinterpolation based on the predicted samples (S951).

In order to derive the input boundary vector using neighboring samplesadjacent to the current block, the video decoder generates an initialboundary vector filled with the value of the neighboring samplesadjacent to the current block or down-sampled values from theneighboring samples according to the width and height of the currentblock, and may remove the DC component from the initial boundary vectorto generate an input boundary vector to which the matrix-vectormultiplication is to be applied. For example, removing the DC componentfrom the initial boundary vector may be or include subtracting the valueof the first entry from each entry in the initial boundary vector. Thefirst entry in the input boundary vector may be obtained based on thedifference between half of the maximum value that can be expressed inbit depth and the first entry of the initial boundary vector, andsubsequent entries of the input boundary vector may be obtained based onthe subtraction of the value of the first entry from each entry in theinitial boundary vector.

To generate the initial boundary vector, the video decoder may decode,from the bitstream, a syntax element indicating the concatenation orderof first entries of the initial boundary vector derived from leftneighboring samples adjacent to the current block and second entries ofthe initial boundary vector derived from the above neighboring samplesadjacent to the current block. The video decoder may generate theinitial boundary vector by concatenating the first entries and thesecond entries according to the concatenation order.

In order to derive the prediction block for the current block based onthe predicted samples, the video decoder may allocate the predictedsamples to positions in the prediction block, and perform horizontalinterpolation and vertical interpolation on the predicted samples, leftneighboring samples adjacent to the current block, and above neighboringsamples adjacent to the current block to generate predicted samplevalues for positions to which to which the predicted samples are notallocated the prediction block. The horizontal interpolation may beperformed prior to the vertical interpolation.

When the current block is a luma block composed of a luma component andMIP is applied to the luma block, the intra-prediction mode of thechroma block may be set to the PLANAR mode if the intra-prediction modeof the luma block is to be used as the intra-prediction mode of thechroma block corresponding to the luma block.

It should be understood that the exemplary embodiments described abovemay be implemented in many different ways. The functions or methodsdescribed in one or more examples may be implemented in hardware,software, firmware, or any combination thereof. It should be understoodthat the functional components described herein have been labeled “unit”to further emphasize their implementation independence.

Various functions or methods described in the present disclosure may beimplemented with instructions stored in a non-transitory recordingmedium that may be read and executed by one or more processors.Non-transitory recording media include, for example, all types ofrecording devices in which data is stored in a form readable by acomputer system. For example, non-transitory recording media includestorage media such as erasable programmable read only memory (EPROM),flash drives, optical drives, magnetic hard drives, and solid statedrives (SSDs).

Although exemplary embodiments of the present invention have beendescribed for illustrative purposes, those skilled in the art willappreciate that and various modifications and changes are possible,without departing from the idea and scope of the invention. Exemplaryembodiments have been described for the sake of brevity and clarity.Accordingly, one of ordinary skill would understand that the scope ofthe embodiments is not limited by the embodiments explicitly describedabove but is inclusive of the claims and equivalents thereto.

What is claimed is:
 1. A method of decoding video data, comprising:decoding, from a bitstream, a syntax element indicating anintra-prediction type of a current block of the video data, theintra-prediction type being indicated from among matrix basedintra-prediction (MIP) and regular intra-prediction; and generating aprediction block for the current block by selectively performing the MIPor the regular intra-prediction based on the intra-prediction type ofthe current block indicated by the syntax element, wherein generatingthe prediction block for the current block by performing the MIPcomprises: decoding, from the bitstream, a syntax element indicating anMW mode for the current block, the syntax element being represented as atruncated binary code specifying one of a plurality of MIP predictionmodes allowed for a width and a height of the current block; deriving aninput boundary vector using neighboring samples adjacent to the currentblock based on the width and the height of the current block; generatingpredicted samples for the current block based on matrix-vectormultiplication between the input boundary vector and a matrix predefinedfor the MW mode; and deriving the prediction block for the current blockbased on the predicted samples, wherein deriving the input boundaryvector using the neighboring samples adjacent to the current blockcomprises: generating an initial boundary vector filled with theneighboring samples adjacent to the current block or down-sampled valuesfrom the neighboring samples according to the width and the height ofthe current block; and from the initial boundary vector, generating aninput boundary vector to which the matrix-vector multiplication isapplied, and where a first entry of the input boundary vector isobtained based on a difference between half of a maximum value allowedto be expressed in bit depth and the first entry of the initial boundaryvector, and subsequent entries of the input boundary vector are obtainedbased on subtraction of a value of the first entry from each entry ofthe initial boundary vector.
 2. The method of claim 1, whereingenerating the initial boundary vector comprises: decoding, from thebitstream, a syntax element indicating a concatenation order of firstentries of the initial boundary vector derived from the left neighboringsamples adjacent to the current block and second entries of the initialboundary vector derived from the above neighboring samples adjacent tothe current block; and concatenating the first entries and the secondentries according to the concatenation order and thereby generating theinitial boundary vector.
 3. The method of claim 1, wherein deriving theprediction block for the current block based on the predicted samplescomprises: allocating the predicted samples to positions in theprediction block; and generating predicted sample values for positionsto which the predicted samples are not allocated in the predictionblock, by performing horizontal interpolation and vertical interpolationon the predicted samples, left neighboring samples adjacent to thecurrent block, and above neighboring samples adjacent to the currentblock.
 4. The method of claim 3, wherein the horizontal interpolation isperformed prior to the vertical interpolation.
 5. The method of claim 3,wherein, before the horizontal interpolation and the verticalinterpolation are performed, clipping is performed on the predictedsamples such that the predicted samples lie between 0 and2^(bitDepth)−1.
 6. The method of claim 1, wherein the current block is aluma block composed of a luma component, wherein, when the MIP isperformed on the luma block, and the intra-prediction mode of the lumablock is used as an intra-prediction mode of a chroma blockcorresponding to the luma block, an intra-prediction mode of the chromablock is set as a PLANAR mode.
 7. The method of claim 1, whereingenerating the prediction block for the current block by performing theregular intra-prediction comprises: deriving Most Probable Mode (MPM)candidates based on a regular intra-prediction mode of each ofneighboring blocks adjacent to the current block and thereby configuringan MPM list for the current block; and deriving a regularintra-prediction mode for the current block based on the MPM list,wherein, when an intra-prediction type of the neighboring blocks is theMIP, the regular intra-prediction mode of the neighboring block isregarded as a PLANAR mode.
 8. A method of encoding video data,comprising: encoding a syntax element indicating an intra-predictiontype of a current block of the video data, the intra-prediction typebeing indicated from among matrix based intra-prediction (MW) andregular intra-prediction; generating a prediction block for the currentblock by selectively performing the MIP or the regular intra-predictionbased on the intra-prediction type of the current block indicated by thesyntax element; and encoding a residual block that is a differencebetween the current block and the prediction block, wherein generatingthe prediction block for the current block by performing the MIPcomprises: encoding a syntax element indicating an MIP mode for thecurrent block, the syntax element being represented as a truncatedbinary code specifying one of a plurality of MIP prediction modesallowed for a width and a height of the current block; deriving an inputboundary vector using neighboring samples adjacent to the current blockbased on the width and the height of the current block; generatingpredicted samples for the current block based on matrix-vectormultiplication between the input boundary vector and a matrix predefinedfor the MW mode; and deriving the prediction block for the current blockbased on the predicted samples, wherein deriving the input boundaryvector using the neighboring samples adjacent to the current blockcomprises: generating an initial boundary vector filled with theneighboring samples adjacent to the current block or down-sampled valuesfrom the neighboring samples according to the width and the height ofthe current block; and from the initial boundary vector, generating aninput boundary vector to which the matrix-vector multiplication isapplied, and where a first entry of the input boundary vector isobtained based on a difference between half of a maximum value allowedto be expressed in bit depth and the first entry of the initial boundaryvector, and subsequent entries of the input boundary vector are obtainedbased on subtraction of a value of the first entry from each entry ofthe initial boundary vector.
 9. The method of claim 8, whereingenerating the initial boundary vector comprises: encoding a syntaxelement indicating a concatenation order of first entries of the initialboundary vector derived from the left neighboring samples adjacent tothe current block and second entries of the initial boundary vectorderived from the above neighboring samples adjacent to the currentblock; and concatenating the first entries and the second entriesaccording to the concatenation order and thereby generating the initialboundary vector.
 10. A non-transitory computer readable medium storing abitstream containing encoded data for video data, the bitstreamgenerated by processes of: encoding a syntax element indicating anintra-prediction type of a current block of the video data, theintra-prediction type being indicated from among matrix basedintra-prediction (MW) and regular intra-prediction; generating aprediction block for the current block by selectively performing the MIPor the regular intra-prediction based on the intra-prediction type ofthe current block indicated by the syntax element; and encoding aresidual block that is a difference between the current block and theprediction block, wherein generating the prediction block for thecurrent block by performing the MIP comprises: encoding a syntax elementindicating an MIP mode for the current block, the syntax element beingrepresented as a truncated binary code specifying one of a plurality ofMIP prediction modes allowed for a width and a height of the currentblock; deriving an input boundary vector using neighboring samplesadjacent to the current block based on the width and the height of thecurrent block; generating predicted samples for the current block basedon matrix-vector multiplication between the input boundary vector and amatrix predefined for the MW mode; and deriving the prediction block forthe current block based on the predicted samples, wherein deriving theinput boundary vector using the neighboring samples adjacent to thecurrent block comprises: generating an initial boundary vector filledwith the neighboring samples adjacent to the current block ordown-sampled values from the neighboring samples according to the widthand the height of the current block; and from the initial boundaryvector, generating an input boundary vector to which the matrix-vectormultiplication is applied, and where a first entry of the input boundaryvector is obtained based on a difference between half of a maximum valueallowed to be expressed in bit depth and the first entry of the initialboundary vector, and subsequent entries of the input boundary vector areobtained based on subtraction of a value of the first entry from eachentry of the initial boundary vector.
 11. The non-transitory computerreadable medium of claim 10, wherein generating the initial boundaryvector comprises: encoding a syntax element indicating a concatenationorder of first entries of the initial boundary vector derived from theleft neighboring samples adjacent to the current block and secondentries of the initial boundary vector derived from the aboveneighboring samples adjacent to the current block; and concatenating thefirst entries and the second entries according to the concatenationorder and thereby generating the initial boundary vector.